One thing I struggled with here is how to balance readability / safety of code vs. performance. Since validation and transcoding are going to be such common operations, I weighted the decision heavily on the side of highest possible performance. This should be fleshed out by the benchmarks when I rerun them.

Many of the more obtuse lines of code exist because they cause the JIT to emit very specific and optimized assembly. I'm hoping that a combination of code comments and test coverage makes the code more maintainable.

/cc @ahsonkhan and @bartonjs who may find this interesting because it'll be followed by the transcoding and escaping APIs you've been asking for. :)

Many of the more obtuse lines of code exist because they cause the JIT to emit very specific and optimized assembly. I'm hoping that a combination of code comments and test coverage makes the code more maintainable.

It would be nice to have a list of these obtuse lines in this PR. Corelib is becoming rather quickly a collection of such hacks and removing them when the JIT improves might not be so easy.

This is still WIP.

Todo:

• Rerun benchmarks over Utf8 comparing it to UTF8Encoding.
• Delete the logic in UTF8Encoding and plumb this class through it. Much of the difficult work was already done in Refactor Encoding to split fast-path and fallback logic #23098.
• Still some pending PR comments to get to.

Removed WIP marker. This is now code-complete. There are some tests failing in corefx since they (incorrectly) assume things about the output of transcoding invalid data. I'll send a separate PR through corefx with those fixes. In the meantime, once I build the full list of failing tests that need to be changed I'll also update the global suppression file in this repo.

Benchmarks are pending.

High-level summary of changes:

• Create vectorized transcoding logic and char counting / byte counting logic.
• Hook up the fast transcoding logic through the System.Text.Unicode.Utf8 class.
• Plumb this logic through System.Text.UTF8Encoding.
• Move internal Utf16Utility type to System.Text.Unicode namespace to live next to Utf8Utility.

@mikedn I've tried to annotate the "hacks" where they appear throughout the code, including with links back to GitHub issues.

Quick benchmarks, using various corpus texts from Project Gutenberg.

These texts are large (~100KB). I'll work on getting benchmarks for smaller texts as well.

Supported commands
     help:
          Get descriptions, examples and documentation about supported commands
          Example: help "command_name"
     run:
          Run all pipelines or a specific pipeline for this repository using a comment. Use
          this command by itself to trigger all related pipelines, or specify a pipeline
          to run.
          Example: "run" or "run pipeline_name"

See additional documentation.

/azp run coreclr-ci (Build Linux_musl x64 checked)

No pipelines are associated with this pull request.

Thank you all @gfoidl @ahsonkhan @bartonjs @mikedn @tannergooding for the thoughtful feedback you've all given throughout this process. I know it was a slog. Your assistance is much appreciated! 🎉

CI failures are due to #23902, which has already been fixed. Going ahead with this as-is.

@GrabYourPitchforks could you squash and merge next time? "Clarify comment in 3-byte processing", "Add missing check to 3-byte processing logic", "PR feedback: Fix typos" and similar clutters the git history.

@MichalStrehovsky I did, including writing an actual real description for the commit, but apparently if GitHub encounters a server-side error during the process it automatically falls back to a normal merge commit. 😒